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FT.EFACE 

The Artificial Intelligence laboratory proposes to continue i1s vcrk on 
a group of closely interconnected jrojects, all bearirg or questions 
about how to make computers able to use more rophistieate* kirds of 
knowledge to solve difficult problems. This proposal e: plains what we 
expect to come of this work, and why it seems to us the west profitable 
direction for research at this ti^e. The core of this proposal is abcut 
well-defined specific tasks such as exterdinf the computer's ability to 
understand information presented as visual scenes, or in natural, human 
language. Although these specific goals are important enough in 
themselves, we see their pursuit also as tightly bound to the 
development of a general theory of the computations neeced to produce 
intelligent procecses. Obviously, a certain amount cf theory is'neeced 
tc achieve process in this and we maintain that the steps toward a 
comprehensive theory in this domain must include thorough analysis of 
very specific phenomena. Our confidence in this strategy is based both 
on past successes and on our current theory of knowledge rtructure. Cur 
proposed solutions are still evolving, but they all seem to revolve 
around new methods of programming and new ways to represent knowledge 
alout programming. The field of Artificial Intelligence has cede 
enormous jrogress in the past few years toward becoming a scientiiic 
subject. This proposal deals with our main goals, both for long-range 

research and icr particular application areas. The general technical 
position of our 1970 and 1971 proposals still represents the direction 
of our approach. However, although these proposals are relatively 
explicit about the high-level problems motivating our research, they do 
net give a very clear picture cf the actual projects, or of their 
practical consequences. In this proposal we shall concentrate on giving 
a more concrete view of our immediate goals. The next four sections ere 
etch directed to an "area" of application. It will be olvious that the 
cencej ts and methods used in tnese areas are closely related. Cur 
intentions about the apjlicatiors vary, naturally. In some cases we 
have i-ajor efforts toward completing effective ojerational prototypes. 
In the rest, we expect to produce demonstration prototypes, or enly to 
develop seme theory, clarify problems, and attempt to direct the 
attention of others to problems which we think need immediate attention. 
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ft\rt J. £ub-I rejects and filestores 

1*0 Anticijated Milestcnes 

We define current and proposed sub-projects 3argely by relating then to 
milestone tests cf achievement which we have set up to male the 
immediate goals and criteria cf our wcrk as unambiguous as porsiWe. Vie 
do net try to predict the date of success for each milestone beyond 
stying that ve would be surprised if any of them toot; as much as three 
years- It would be best to 

legin by discusring a milestone to whese achievement ve are finely 
committed because ve are sure that it is in the state cf the art end 
because it will have impcrtart theoretical and industrial consequences* 
It combines in a very interactive — i-ore than merely additive! — way a 
ltxge number of theoretically related sub-systems that have, J or 
practical reasons, been built up separately* 



1*1 The Language-Virion-Action Eemonstration 

This milestone will be passed when we can give the computer (equipjed 
with eyes and hands) orders in brief, business-like language such as 

"I think there is a defective diode on this 
circuit board* Probably the third in the row en 
the left. Check and replace if necessary." 

Fcr this test to be meaningful, ve must assume that 

— the computer has no prior knowledge about the particular circuit 

board (e.g. in the form of a diagram or symbolic description). 

— the computer is able to see the circuit beard by ratural vision 

(so, for example, there are no special marks en it like the 
u-gnetic characters on checks)* 

— the English expressions used really are free (e.g. there is no 

hidden format beyond the restriction to what a real, rot 
particularly smart, human technician would accept as © straight, 
natural ar:d clear instruction)* 

■ ■ 

The achievement of this demonstration dejends on realising a nurater of 
sub-gcals of which some depend on ports of what is seen in the 
demonstration itself while others are more like the oil ir the machine — 
nc less vital and difficult fcr being invisible to the nrive observer. 
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These sub-coals include: 

— cxtencing the domain cf t! e vision system fror. blocks with plme 

sur facer tc the mere natural objects found in electroric 
coraponertry with their curves, highlights, colors, stripes, 
shacows and so on. 

— corresponding extension of the manipulative ability. 

— extending the natural language pFGgnius ir several directions: 

— new domain (electronics instead of blocks); 

— modalities such as probtble-posrible-necesscry; 

— modelling the person giving the instruction, so as to deal 
properly with such information as "I think**-" 

— further evolution of programming languages, computer control 

structures and cebufginf aids, in fcrms particularly suitable 
to work in Artificial Irtelligerce. 

— theoretical problems in the area cf representing knowledge* 

including "Leanings' 1 or "world models" constructed from visual 

and linguistic inputs. 

■ 

This rough breeze own into sub-goals illustrates ore of the ma*or 
theoretical theses of our laboratory. An "intelligence" is a complex 
system with a larfe number of interacting but serarate parts- It can be 
understood and reproduced only by dealing specifically with these parts* 
t'e do not believe in the chimera of finding a single powerful "tiethcd" 
cr "principle" which would give rise to intelligence the \:ay the laws of 
Nevtott give rise to elliptic crbits. In surveying the variety of 
speciiic sub-projects mentioned below, the reader is entitled to be 
cencemed about how many sub-problems we anticipate having to sclve in 
order to achieve new levels of irtelligence in automata! Is it tens? 
hundreds? millions? Arc we merely scratching the surface? One of the 
purposes cf unified milestone demonstrations (such as the one under 
discussion) is tc show that the numbers can be contaired. Speaking 
informally, we believe that the number of subsystems of intelligence is 
large, but net as large as the number of subsystems engaged in, say, an 
Ajollo mission. The topics that have already been considered in the 
laboratory constitute a significart fraction cf what is necessary. 
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1.2 Other Milestones: VISION end the ILCCKS WCRLD 

We will describe the other milestones mere triefly, though they are no 
less serious. Soiue of them are easier and will be achieved earlier than 
the Lanfuape-Visien-Action der.ons'ration in electronics; some- are much 
mere difficult. 

The BLOCKS WOKLD has jlayed a central role as a culture nediuir for ideas 
about vision and will continue to do so despite the new concertration on 
other nicre useful and cocplex areas of work. To place new milestones in 
perspective we recall briefly some past history. 

The first important steps toward the present approach were the 
work, more than a decade ago, by H. Exnst on touch-sense controlled 
canipulation and by L. Roberts on machine scere-aralysis of 
photographs of three-dimensional plane-surfaced cbjects. 

The next significant step, this one under AKPA suppcrt in the A. I* 
lab (then part of Project I* AC ) was the development of visually- 
controlled tower-building programs in 1965-1966, The majcr sub- 
system was direct "natural" vision of real, but unoccluded and 
clearly illuminated tlocks. 

In n.ore recent milestone demonstrations, ve have seen machine 
manipulation of several tlocks in visually occluding relationships. 
Fehird this practical step vas a mere important theoretical ore, 
first taken by Guzcan, of basing scene analysis en peneral 
knowledge about "bodies" or "objects 11 without usinf (as was dene 
before) specific knovledge atcut particular kinds of bodies. 

In this older work on vision, shadows and other side effects cf lighting 
were treated as emtexrasring cocplicaticns to be minimized. However, 
these effects are sources of inforjnatior, and we now urderftand them 
well enough to exploit them as such. A milestone for the very near 
future will be a Vision Systen to see highly shadoved and badly 
illuminated scenes, with the help of (rather than in spite of) the 
shadows. In the Ltnguage-Visicn- Action demonrtration, the relation 

between the hand and its shadow should be exploited to anticipate the 
ccntact of the hard and its target.' 
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The KIT VISION SYLTEM is, at parent* still an experimertal prototype 
developed to explore new ideas both rbout machine virion end new ety3.ee 
of programming - It as the tost powerful sy&tei availal le within its 
narrow domain of ar;alyr ing cor ocular, polyhedral scenes, V.'e ore 
extending its capabilities to deal with rcore natural objects. Up to 
new, this system counld deal only with jx>]yhedral objects having unifcrn 
plane surfaces:, such its blocks* wedges, KTamids, and the like. 

Vision programs generally use information of just ore kind: light 
intensities on a two dimensional prelection- An important imminent 
advance is the ur*e of multiple rources cf information. Sore of these 
will te other dimensions of vision — color, range, etc. Other rources 
wil be symbolic, in particular these derived from irterrcticn with 
StJtDLU* We do KOT see as a significart milestone merely usirr top 

level commands in English, parsed on through SHRDLU, to operate a Vision 
Svstem. This msy be a good demonstration to the outside wcrld of the 
flexibility ard utility of SHRDLU, but it is really quite easy. The 
much more significant step will be to carry SHRDLU and VISION sc as to 
permit statements made in English to give useful advice to the vision 
program: "Look more carefully in the shadow of the cute to the right 

of the tall block* 1 . A much more difficult stej — sufficiently 

significant to count as s separate milestone if done in an insightful 
way — is telling the program in English how to extend its mini-world of 
competence. Por example, one might tell a program desigred to see and 
manipulate blocks about a good strategy for putting things in boxes. 



1.3 Vision Outside the BLOCKS WORLD 

Liquids: An excellent test-bed for ideas about changing shapes. The 
milestone is an eye-hand system that will make a cup of regular 
instant coffee. Tnis is described below in more detail, 

Fecple-Watching: Cur own research on the control and acquisition of 
tctor (and postural) skills in humans will benefit from a 
program capable of observing a person engaged in a task such as 
learning to walk a tight-rope. A simple application of this is 
detecting when a person approaches dangerously clcse to the edge 
of a platform, and warning him. 

Interpretation of Drawings: Hie interpretation, fey machine, of net- 
quite realistic or conventional drawings peses problems 
intermediate between vision and language. It is a tarlc that has 
net yet been performed significantly by machine, yet seers ripe 
as a future milestone. We hope, through work like that of 
Goldstein (see below), to develop a program that can describe in 
wcrds — that is, by understanding— what is shown in a drawing, 
cartoon, cr action-sketch with stick-figures. 
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1*4 Understanding English 

Our Laboratory has pursued over the years a working hypothesis that can 
be stated briefJy as: before cne can pet a machine to understand 
English, cne &ust find hew to make it understand at all, Trarslated 
ir.to concrete terms, this means caking programs to understand complex 
English statements abcut a simple nini— world which the computer is atle 
to understand very thoroughly by draving on a stack cf special 
knowledge. An early miles tore whose achievement encouraged this point 
of view was the early program, STUDENT 9 by Dan Eobrow. The latest clear 
break-through is tKRDLU by Terry Unograd. 

A secondary cilestone is the operational use of SKRDLU bs a "front-end** 
to con&unicate with programs written for quite different, practical 
purposes. An early example of this kind of use is the coupling of 

SHRDLU to a question answering program made by CCA. We know of several 
other such projects- Though it is encouraging to cee Artificial 
Intelligence programs being used core ard mere extensively, these uses 
of SHRDLU do not go beyond its original fcroal capacity • Mere 

substantial ralestones which we ree as significant end yet not far eff 
include : 

Using frore Linguistic Information: A program that makes 
significant use of tenses, modalities (e.g. posrible-prcbable) 
and indirect reference (e.g. He thinks that ) would 

be a considerable advance. 

Getting Away With Less Linguistic Information: We have in mind 
using "common sense" interpretation to fill in only partially 
specified information. For mere precision, see "Narrative", 
below. 

Extending SHTJ)LU in English: A most important milestone will be 
passed when we are able to describe, in English, extensions to 
SKRLXU itself. SHRLLU's interpretational power would then act 
in a bootstrapping fashion. This goal is more inclusive £nd 
more difficult than the others wo have listed. 



1.5 Computers, Knowledge, and Intelligence 

The following sections describe what we propose to do in much greater 
detail. Before entering this feres t of detail, we want to explain cur 

in<age of how this approach relates to others. 
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The world of Science is well-crgarizec for the management of rome kirds 
of krov.ilc.dge* Jts thecrier provide a firm (though net, of course, 
infallible) understanding of what kirds of know] edge are relevant to 
such tasks as predicting astronomical everts or designirg bridges. 
There are institutionalized repositories of such knovledge (handbooks, 
tables, encyclopedias) as well v.s the means of tranrcitting it to the 
next generation l&chocls). 

Set ^n knowledge has been treated in this fora-al Earner* In 

particular, a very large body cf knowledge has traditionally been 
neglected for the simple rearcn (hat "everyone knows it anyway" • This 
is the kind of common sense knowledge that leads one, for example, to 
rearrange the contents of a box (or throw something out) when more spece 
is needed. 

Research in Artificial lntelliger.ee as forced to deal with this kind of 
knowledge quite explicitly* One eight argue that the "intelligent 
computer" ought to acquire such knowledge by the tacit, informal process 
that leads humans to have it without explicit formalization. Ihis is 
certainly true in soLe sense; but even to understand what is being 
asserted ve need to formalize i~cre explicitly, and characterize more 
insightfully, the kinds of knowledge in question and the kinds of 
processes that Eifht lead to its acquisition. 

Among our (human) ways to acquire knowledge, two stand out beyond others 
and our work has centered on their; Language and Vision. The rest of 
this propcsal is divided into three parts, accordingly. 

Part II is concerned with issues related to logic, general 
knowledge, and common sense, in a context centered mainly around 
problems of understanding natural language. This section also 
includes closely related wcrk on understanding procedures, 
programing languages, and descriptions of programs* 

Part III discurses issves connected with robotics, machine vision, 
manipulation, and other activities that involve physical-wcrld 
interactions- 
Part IV discusses work needed to develop tools: hardware, 
programming languages, software*, and other things needed to 
support the goal-directed projects. 

These areas are, of course, not reelly separable; all of them, for 
example, have sut^areas that work with knowledge about the EICCKS VORID, 
a simjle model physical world of simple, easily described objects and 
interrelations, that serves as well as the subject for logical, 
linguistic, rcbotic, and program- theoretical studies. 
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er of projects ax the Laboratory are certered around the problems of 
urderetanding natural language. There studies ircluce bcth theoretical 
and practical prollemr: 



2-1 Expert Version cf Vinogradov Langua^e-Understander 
Terry Wincgrad and group 

2.2 More Powerful Problec>-Sclvirg in SKEDLU — Using the 
CClfKIVER language Instead of the Erlmitive liicro Plnnrer SystcK 
1,'inograd and grou 



our 

. Pi 



V« Martin and G. Erovn of MAC 

2.3 Hew Models for Meanings 

■■■ ■■■ Kinsky, Killer f Charaiak, and others 

2.4 Hew Grairciar Organizations 
D. McDonald, A, Rubin, V. Pratt 

2.5 Theories of Generic Expressions and Typical Situations; 
Scenarios, English t Cuantifiers, and Logic 

■■ — R. Mcore, M. Marcus 

2.6 Learning to Understand 
t. McDert\ott 

2-7 Understanding Stories 
E. Chemiak 

2.8 Sussman 's Program: HACKER 

G. Sussman 

2.9 Knowledge about Procedures and Their Likely Pugs 

-■ ■ ■ G. Sussman 

2.10 Goldstein's "Solution" cf the Halting Problem 
— " ■■■ ■ I. Gcldstein 

2.11 Goldstein's Program-Understanding Program 
j, Goldstein 
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2.1 lolichinf the Larruafe-Urderctancing Prorraffl 

A project that will be completed this year is the conversion of 
Winograd'r l&nguf-ge-understandinf prcgrai: (called SKRELl) froia a first 
demonstration prototyje into a generally accef^ible, vell-documenled 
experimental facility for further develcpmert cf linfurtic models and 
applicaticns that are related to the basic theory behind this model. 
The new system contains extensive tracing and debupgirg aids, and a 
detailed i-anual is being prepared. This vill rake the system accessitle 
to users who begin with very little prior knowledge about its details. 
The system is available over the ARPA network. An early version of the 
EEjuial was prepared last sunnier with the help of Stuart Card at 
Carnegie-kellcn, and we are now adapting it to provide a full guide to 
the system. Current sections include: 

Erief Description of SHREW 

Implementation and Version Information 

The Distribution Package 

Instructions for Running SHRLLU 

hone Pacts about the Program 

References fcr Kore Information 

Examples of Using the Features of the System 

The system has many features for making it ea^y to develop and debug 

extensions, and specializations fcr particular applications. These 

induce a self— explanatory command tree system that allows users to view 

various parts of the system in interactive modes. 
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2.2 More Powerful rroblem-Solving in SKHDLl 

Several students are rewriting parts of the language understanding 
system to use CCKNIVER instead of hicro-Planner* The new language, 
which permits cress-reference between different parts of a problem 
solving process, will provide a letter base for ex tend tap the langurge 
program to a larger world that car use hypothetical contexts, plausible 
reasoning, and more complete handling of tenses. 

We are also working with other froups in their uses of this language 
understanding syslem* The project of W. Martin, at Project MAC, will 
probably use it as a Eeans of communication with their automatic 
programing systems, and G. Brown, also at MAC* is exploring the 
possibilities of basing a language translator on SHRDLU; at present she 
is working on a very small specific set of problems involving Geman 
ncun cases and prepositions, but the basic ideas Bay be generalizable* 
In any case, by writing a comparable grammar for German, she is 
increasing our understanding of which basic mechanisms are best ior 
representing natural language grarmars in general* 
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2.3 New Hoc els for K earing 

At a theoretical level, several of the staff and students- are exploring 
scce rew formalisms fcr representing mearings. One problem is to tie 
together a variety of phenorena that have been addressed mere cr less 
separately irj such models as Fillmore's Case Graoncrs, Abelsor's 
"nolecules , texts, scripts, etc.", Schank's Conceptual Grammars, 
Kalliday's Systemic CramiLars, etc. We believe that these can be tied 
together in a more consistent way by viewing language understanding as 
an active procedure, constantly engaged in an operation of "fitting 
together" inputs and the implications of inputs, Ihese theories are 
just beginning, but ve hope they will lead to a second generation system 
much better equipped to handle plausible reasoning, incomplete inputs 
and a vider range of ways in which language conveys meaning. It is too 
early to describe the theories in detail, because they do net yet have 
adequate formalisms, but among their new elements are what might be 
called "scenario?" — structural elements that describe "the ways things 
usually happen" — that are subrtantially larger than the kinds of 
elements found in earlier syntactic or semantic theories* 
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2.4 I*ew CratLiar Organizations 

Ctvid Hctcnaid is working en problems of generating sentences that 
represent Eiear.ingr in ways that are rerponrive to the needs cf the 
system's user- Ihis mears that the systen must use its knowledge abcut 
what (it thinks) 'he user believes, Wincgrac's original SKRILU system 
does this to a certain extent; it takes into account sccie cf what the 
user can be deduced to know when it answers questienr. McDonald's 
approach to generating coherent discourse views it as concerning 
multiple procedures operating on a cocmon data structure- First, there 
is a logical process concerned with choosing words and producing 
structures which convey the underlying cleaning* Second, there is a 
discourse procedure vhose jot is to structure the output for coherence, 
taking the necessary inter-sentence connections, introducing pronouns 
when appropriate (and when their referents can be determined, 
presumably)* Third, there is a syntactic specialist to make sure the 

output is in a proper grammatical form- McDonald KBiita to avoid the 
kind cf inflexible organization in which each of these is forced to work 
on a completed output of another. Thus, he wants a heterrrchical system 
(see below, in connection with Vision and Robotics) in which they titie— 
share, each being allowed to Dake suggestions as to how the output 
should be formulated, and reacting to suggestions of the others. The 
goal is to product an cutjut systefi for use with the £KRDT.U program with 
capability couplet en tary to that rystem's input understanding ability. 
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A. Rubin is working on a new version of Vincgrac's £ramrar, exjloring 
different tyres cf organization which pet avay froc. the highly linear 
organization cf the current grnoiaar. Kore efficient and flexille 
grammars may take Tuore syntactic cues from the words themselves, 
resulting in a still primarily lop—down parser whore operations rre 
primarily initiated by bottom-up methods* Herds such as "a" and "the", 
or prepositions and question words have ijiBcdiate syntactic irplicaticns 
and should be able to direct the parsing process. Reliance or the words 
themselves instead of their pesition in the utterance should te a great 
help in parsing incomplete and unrrammatical utterances, especially when 
ccupled with a coherent disccurse content. In fact, such exploitation 
E£y be sicply necessary fcr dealing vith real-life interactions, Eany of 
which are fragmentary yet perfectly intelligible. Rubir is preparing 
Howcharts so that the resulting grammar can be used both for 
constructing programs and for describing and studying English. People 
with limited knowledge abcut FRCGFiAKMAR and LISP (the computer languages 
used by the system) but who have substantial knowledge abcut linguistics 
will be able to use these charts tc get access to a computer-based 
grammar fcr a substantial part of English. 

■ 
We hope this will accelerate interdisciplinary communication and 
progress, and this seems likely because there is already world-wide 
interest in this system. The flowcharts should make it possible for 
people unfamiliar with details of the profraiE to experiment, modify, pnd 
extend the grammar for their own applications. 
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Rubin is also studyirf problems cf extending the syrterc to use complex 
tense end time semantics; perfect and progressive tenses, future?, rnd 
Exdalf . Tenre ar-d other time components combine in rct-vell-understcod 
vtys; compare: f, I see ry adviser Thursday" with "I see ry pdviror 
Thursdays 11 ; some deduction is required to decide which sense of the 
terse is indicated* Compare "The car runr on gas", which is essentially 
tenseless, with "the car needs to be vashed". To deal with these will 
require charges in both semantic and deductive parts cf the system, 
using new ways to represent pfist, future, and hypothetical assertions. 



2.5 Theories of Typical Situations and Generic Expressions 

R- I oore has been exploring some of the protlens in connecting English 
expressions to their underlying logical forms. He is concentrating 
particularly on Enflish quantifiers and their connections rath 
mathematical logic. The connections are not straightforward, and one 
dees not often "mean" a quantifier in the sane rense of the 
conventionally equivalent logical symbol. A word like "any" can teve 
several different meanings in logical terms. In the sentence "Fid 
anyone cote to see me?" it corresponds to the logical "there exists" end 
we can represent it as (Exists X)(X came to see me)« Fut in the 
sentence "Intercept anyone vho cojnes to see i,.e. " it represents a 
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universal: (I'orall X)(If X comer to see nic, intercept X)* Mcore is 
vcrkirg on a formalism tl^t coml ines predicate logic with a sort of 
lattda calculus vhich can deal with this sorl of pherorceron, as veil as 
mere conplex problems involving embedding and pronoun reference* 



H. Karcus is also studying questions that seem closely related- What 
are "generic" nouns and what is their role in comor.sense reasoning? 
When cne says "A bird can fly 1 ', ore legical interpretation is that this 
neons "If X is a bird, then X can fly". We don't believe thet this 
traditional interpretation is correct enough to be usable* Some birds, 
like the ostrich, can't fly in an.y case; other birds can't fly row 
because they have broken vings, are tethered, etc., etc* So cne usually 
means something like "typical birds fly" cr "If X is a typicrl bird, X 
c&yi fly' 1 * What dees this Bean, in turn? 

Cre direction many workers have explored is that cf formalization within 
other logical schemes. V/e believe that this is net entirely a matter of 
formalization, however J Perhaps there is a such deeper issue here, cne 
of content. The understanding cf what is "typical" depends on one's 
knowledge and familiarity about the particular subject matter* V/e think 
that useful solutions to understanding such statements ere probably to 
be found in a system that uses "scenarios" of the kirds cf things that 
"usually" happen, and that, hence, the interpretation of statemerts 
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atout "the average t:an" and the like are to be iased or detailed 
"general" kncwlecge about the subject rather than on the discovery ol a 
new lexical quantifier or rule of inference- 

■ 

2.6 Learning to Understand 

D. McDerjLott is developing a broad, tut rot deep, prcgrair called TOHE, 
which tries to understand simple declarative statements about a world 
like that of Wir.ograd (1971). TOPLE can "visualize" certain spatial 
relations, guess caudal explanations of simple behavior, and make 
predictions about the future course of a sequence of events. It has a 
limited atility tc understand the actions of other creatures, and mke 
very simple models of their states of Kind. This research is addressed 
to the problem of developing programs that can be told new things 
instead of having to have them pains takingly programmed in* It has 
often been suggested that this could be dene using a set of programs for 
translating declarative statements into deductive assertions and 
theorems. There are several syntactic and semantic reasens why this is 
net sufficient; 



1. The program will inevitably be seeing new words end phrares 
as we talk to it- It must have rophisticatec ways of translating these 
into formal language. It is proVaoly possible to guess the syntactic 
category of a new word, or even phrare, from its context (Thorr.e, 
Eratley, and Dewar, 1S6fc), tut that is only the beginning of the 
problem* If "pen" is a relatively new word (whose meaning* however, has 
been told to the program), it must be clever enough to avcid translating 

"The Ean shot the pic in the pen*" 
the same way it dees 

"The man shot the pig in the heart." 
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2. Fven when completely stupid interpretations are discarded, 
many sentence;; cor tain syrtactic and pemartic ambiguities that carnot be 
resolved without fuesf-lng which interpretation is most plausible. lor 
example, without taking context into account, it is impossible to say to 
whom "him" refers in: 

"Her father hated Ion, but she loved him aryway. " 

3. fcumar lanruafe is abtreviated. Feople very cfter leave cut 
steps in arguments ard stories, explanations for events they recourt, 
and qualifications of sweeping statements, when they thirk these can be 
reconstructed by their listeners. In human communication the speaker 
usually sketches the situation he is describing with an few strokes as 
pcssitle. It is up to the listener to fill in the r.ost plausible 
details consistent with what he says. If he says, 

"The club crew. Scon they needed a bipger clubhouse." 
no human would fail to see the causal progression the speaker intends. 
This reconstruction must be applied to aloost everything the machine 
hears, since an English sjeaker often leaves unsaid information 
regarding time, causality, and situation, all these being implicit in 
what has gone before and what he is most likely to mean. 

4- Human beings are not infallible. They can be inconsistent, 

misinformed, or unclear. Two people may disagree, and seme people even 

tell lies. In any such case — when imagining the situation being 

discussed is impossible or very costly — the machine must refuse to 
believe in it. 

KcDermott's work on problems 2 and 3 have influenced the development of 

the CCN1.IVER proframiuing language, and the development of TCPLE, which 

is written in CONKIVEfc. Farts of the system are already prograoned. 

So far, the system cannot absorb new COKMIVER programs, but consists 
acstly of a set of carefully tuned prograas which take responsibility 
for adding new statements to the data base as well as retrieving 
conclusions from what it already knows. These programs constitute a 
belief system (or, in Abelson's (1973) terminology, a knowledge system). 
which is committed to making as much sense as possible from things it 
hears. Ihe programs (called "if-r.eeded methods") which embody its 
knowledge of the world, possess some knowledge about what things ere 
easiest to believe, what contradictions are best to wcrk en, etc. TOILE 
does bookkeejing, but lets these experts dc mest cf the analysis of 
statements it hears. (It dees not handle natural language directly, but 
accepts statements in a predicate calculus-like fcrnat thst is directly 
assimilable by the pattern matcher that calls if-needed metheds.) These 
methods ponder questions about the possibility and plausibility cf the 
things they hear; communicate their difficulties back to their callers, 
which can give advice or try something else; and are free tc construct 
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hypothetical worlds to. test theories about what the world reslly locks 
lake, or vhat a sj eaker really me^nt, Given what Ihey are told. 

Thus, for example, the rcutire vhich accepts statements about physical 
locations attempts to find the mo&t likely arranrement of objects which 
is consistent with the locations fiver fay a speaker* The routine which 
handles statements alout poire j laces recofnizes the vagueness with 
which j-ecple specify destinations and attempts to fipure out the mcst 
likely destination that the crealure it is tcld about cculd "be aiming 
at. (For example, "he went ever to the chandelier," usually means, "He 
went ever to the iart of the floor directly under the chandelier. 11 ) 

Basically, TOPLE vill be a very skeptical program, which tries to resist 
changes to the data tese, chocsinf the least jarring charges when they 
are enforced by its being spoken to. Since its programs are doing 
plausible reasoning instead oi airtight deduction, it is essential that 
they reccrd their reasons for the beliefs they add to the world model, 
and that these reasons be accessible to any program thct later cores 
into conflict with them. These reasons are stcred in a very general 
fashicn as programs which hagfle with the newcomer abcut vhat the world 
is really like. The current verr ion of TOPLE, when finished, will be a 
substantial extension in our understanding of orcblet— solving with 
incomplete data by "coiamcnserse" rearonin^. If all foes well, a later 
version Of this program should have linguistic knowledge and could be 
attached to a system like Winoprad's SHRDLU parser and semantic 
routines. There is a broader problem here that should be attacked: if 
a routine is to understand people's speech fully f it should understand 
how language is used as a tool, vhat people are likely to want to 
accomplish when they say something, as well as what a sentence would 
tcan taken completely out of context. Until this sort of knowledge is 
formalized, machines will be deficient in lanpuafe understanding and in 
fully understanding a person's motives and intentions from what he says, 
and hence vill mirs much cf happens around them, since most of what gees 
or in stories and in real life is dialogue. 



2.7 Understanding Stories 

E. Charniak, whose thesis on rejresentation of knowledpe in children's 
stories was completed last year, continues to work on problems of teyt- 
ccitprehension. Charniak believes that the best way to construct a 
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theory of krowlcdge is by exhaustive analysis of rxrticular stcry 
fragments in an attempt to pin down exactly wh2t facts ere needed in 
order to understand the story, and to see hew these facts ought to be 
accessed. Charriak begins with the model preserted in his thesis, in 
which a variety of "demon*" cr jremi-autcnoiccus programs control 
inference and information retrieval when "set" cr "activated" by 
characteristic words or events in the story. 

When one studies knowledge at this derree of fine detail, one encounters 
a great many problems. Because there is not nuch systematic theory yet, 
we will illustrate the situation by example. 

In Charniak's thesis occurs the foil owing example to illustrate seme 
problems in determining ncun phrase reference: 

■ 
■ + 

(1) Soday was Jack's birthday. Fenny and Janet went to the store. 

They ^w ere going to get presents. Janet decided to get a top. 
Ton t do that" said Fenny. "Jack has a top. he will make you 
take at beck." * 

The problem is to recognize that the "it" in the last sentence does rot 

refer to the top Jack currently owns, but rather to the ore Janet is 

thinking of getting. In the thesis, it was suggested that a rule of the 

following form might be at work. 

- 

(2) If we see that a person P eight not like a present X, then look 

for X being returned to the store where it was tought. If we 
see this happening, or even being suggested, arsert that the 
reason why is that P does not like X. 
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Charniak continuer; 

"While the exact Torn of the rule was not crucial to the rrru^ent 
in the thesis, I heve f iven sone further thought tc the prcblec of 
exactly what irforr-Gticn is at work in (1). ly first corclurion 
is that (2) if not the piece cf information which is used in (1). 
Consider: 

(3) Today was Jack's birthday. Penny and Janet went to the store. 

They were going to get presents. Jf net decider to get a top. 
When fhe told Penny this, Penny said, "Jack will make you trke 
it back." 

In this story, prior to Penny's statement we had no reason to believe 

that Jack did not want a top. Cf course, in this story there is no 

problem with pronouns, yet, we understand Penny's statement as irplying 

that Jack does not want a top as a present, and that is the reason why 

the tcp will te taken back. Hence, a ndniDUE modification of (2) would 

be (the change is capitalised): 

(4) If we see that a person P kIGHT GET a present X, then look for X 

being returned to the store where it was bought. If we see this 
happening, or even being suggested, assert that the reason vhy 
is that P does not like X. 

"Proceeding in this way we can construct many variations en (1)* 
all of which utilize the fact that (2) represents* However in 
each case ve can construct the story so that (2), as written, does 
not apply. The goal of all this is to find a "correct 11 version of 
(2) which will account for all of the examples." 

Charniak is also working on a very different probleia-solvirg domain; 
problems in electrostatic field situations. In doing this, he is 
returning to attempt to apply new problen— solving rethods to applied 
tihysics and analysis, the subject of his Master's thesis in which he 
attempted to extend the work of Be brow to irore realistic physical 
problecs and found that the problems of ordinary cOEnionrense reasoning 
pesed critical obstacles* 
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2.6 £usscsn*s Program: h'AC/.'ER 
Consider the piece of advice: 

"If Ihe goal is to place Elock A on Elock E, first 
checfc to see whether there is erough sotce en I; 
if there is not, rearrange the blocks already on E 
ly strategy aiDYUF ard try again." 

Knowledge of the sort carried "by this advice is successfully used by 

A.I. prolans working in a very simple wcrld of tlocks. However, the 

advice is needlessly particular. The strategy suggested is obviously 

net special to blccks; one certainly ought to be able tc forculate it 

as a statement about the allocation of space in more general situations. 

But even this is too particular, if the wcrd "space" is taken in a 
literal sense. Consider how one uses the spatial netaphor in talking 
about computer memories. This reflects the general usefulness, in 
thinking, about the abstract organization of memory, of "spatial 
strategies". That is to say, that information in memories can often be 
managed by using the same "trickr" that work for handling objects in 
real space! For example, rearranging (compactifyirg) non-contigucus 
blocks of data can create space fcr a new block of data. 

■ ■ 

Gerald Sussman has translated these very general remarks into a quite 
new paradigm for a program that Jearrs from experience. Sursnan gives 
his programs access to two different kinds of knowledge: 
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?jiowi edge that really is rpecif ic to tl e 
particular dccair {fcr exaeple, e specific s^t cf 
colored wooden blockr), 

Knowledge at out strategies (fcr cxamj-le, fcr 
allocation cf resources, or ror the ordering cf 
^ub-task* ) elated in a fcrm nuch Eore general then 
that in which they will te wed* 

Sussman's ingenious idea is to rej resent the general know] edge in a fcrm 

that can be expended into actual dec^air -specific computer code when 

needed. This new code chows as an addition (patch) to a program that 

had run into an error condition (rhowing the presence of a bur, ar ^ thus 

the need for a patch)* The addition of the new code can be thought of 

as automatic debugging of the old program! 

Consider a concrete example that Su:rsmar has used 
to demonstrate the potentialities of his idea. He 
begins with a program written to place block A en 
another block, E, This jrogrexi works perfectly as 
long as B is clear of any obstruction* How 
suppose that the program is asked to place /. on E 
when there is already a block C en E, 

In the usual context the program would fail, 
perhaps with an error comment. In Sussman's 
context the error comment triggers the generation 
cf an extra piece cf cede which causes B to te 
cleared "before A is placed on it. The jrogram is 
row expended into iz more complex form and will 
stay that way until it once more falls into en 
error condition revealing a new bug and leading to 
another patch, 

Sussoan's demonstration makes it very plausible that quite complex 

programs could be grown in this way ty successive cacro-exparsior of a 

small number of general strategies* Sussmar will pursue this goal as 

the topic for a Ph.D* thesis which will be completed during the coming 

year. 
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£»9 Knowledge A tout Procedures and Their Likely Bugs 

Tc apjreciate the thinking behind HACKER cne r:ust realize how misleading 
it is to think that Sussftan uerejy replaced the particular ly the mere 
general* Any particular stal ement can be per era3 ized in many 

directions! Which shculd be chosen? A working hypothesis of much wcrk 
in A.I- is that the most powerful generalization in exes* such as these 
terd to take the fore of properties of procedural structures. The 
meaning c f this will be filled cut by examples from on-going projects. 

The first is taken frcm HACKER itcelf- 

. 

Suppose that the program we referred to before is presented with a scene 
containing a blue block, a freer block and a pyramid* The pyramid 
happens tc be on the green block. The program is asked to put the blue 
block en the freer block. It observes that the top of the green block 
is filled; it knows that the remedy must be to remove the pyramid, 
which it does, but puts it on the blue block- On the assumption that 
only single objects can be lifted, this is a disastrous ret; it cleprs 
the green block but makes it imporsible to move the blue. 

JJow the question is: what advice could save the program? On the mere 
specific level one mifht try: 

if you want to put block A en block E, avoid 
jutting anything on block A. 
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Expressed (as it easily cculd be) ir ar appropriate jrogramming 
language, this would save the situation— tut we'd prefer rocethirg mere 
general- What? 

Ore possible direction of choice r.ight be to federalize "Tlocl:" ard w cn" 

to "objects" and "relations" saticfying such and r,uch axioms- Certain 

contemporary approaches to A.I* vould go in this direction despite the 

difficulties encountered in working with appropriate axioms, KACXER 

actually takes a very different direction of generalization. According 

to Sussman, the problem is not about objects and relations but about 

programs — and particularly about a kind of bug that typically arises 

from interactions between attempts to satisfy a sub-goal (clear E) £nd 

the conditions (A being clear) still holding at the time the super-goal 

(neve A to B) was set up. To formulate the advice properly, he takes a 

very general powerful addition to the programming language: an 

operation called "protection" whose effect is to inhibit changes in 

aspects of a situation to which it is applied. Using this operation, 

the advice about not putting block B on block A turns into something of 

the fern:: 

Protect all known recersary conditions for the 
super-goal before examining a suV-goal. 

This transformation cf advice atout blocks into advice abcut program 

structure (goal— sub-goal interactions) illustrates an expanding theme 

of research in the Laboratory— of which KACKER is a particularly clear, 

but net urique, example. To see the cocmon eleaent clearly with the 
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next example, recall that the protection advice derives from 
reorganizing the likelihood of a certain form of interaction tug. 
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Coldctein's "Solution" of the Halting Preble* 



appose you have written a program, P, which is intended to make a 
certain decision and then stop. You suspect that a bur causes it to 
fan into a never-ending loop, i.e. not tc haJt. It would fce very 
veiualle to have a program called, let us say, SuTERMCKITCR, which would 
examine program P and tell whether or not it will halt. 

Everyone who has taken a course in automata knows that an infallitle 
version of SUFERMGKITCR cannot exist— the halting problem is unsojvable' 
Given any alleged version of SUPERMOHITOR, it will be possible to find a 
program P whose failure to halt is too tricky for SUPEiil-lCl.ITOR to catch. 
Eut while programs can, in principle, halt or fail to hclt for 
exceecingly complex reasons, in real life many (possibly "most" or 
almost eU"J accidental causes of failures to halt are very simile 
indeed. Thus the undecidability theorem in no wav precludes the 
£f^mnnT& D cf E Poetically useful (though fallible^ version of 

Ira Goldstein, also a graduate student in the A.I. lab, has chosen as 
his thesis topic the design of a monitor which could be thought of as 
looking over the shoulder of a novice programmer and raking comments 
atout why and when his programs work or do not work. Cne of the 
functions it will perform is detecting inadvertent infinite loops. 

To appreciate the wide applicability of such work, notice that the hucan 
novice programmer cculd himself be a computer program generating 
programs. Thus, besides providing e clear paradigm of the concept of 
"teaching machine" and new insights into the nature cf programming 
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krowlf;dge, Gcldstein's wcrk is directly relevant to all areas in which 
programs are automatically gererated. 

Like £ussr.an f Golcstein provides his program with krowledge about the 
programs peoile (and machines!) actually write and the troubles they 
eight get into. These projects are complementary in corcentrating on 
disjoint aspects. The following example illustrater the aspect 

Goldstein studies in this pirt of his work; in the next section he 

discusses a very different aspect of programming knowledge. 

+ 

Consider a simple exercise program that might be written by a beginner, 

(The program is ir LCGO.) 

50 COUNTDOWN :NUKEER 

10 PHINT :NUKBER 

20 COUNTEOWN :NUKEER-1 

The effect of the contend COUNTDOWN will be to print 
*» 2, 1, 0, -1, -2 f . - . and so on in a loop which would never end 
but for accidental limitations of the computer used to imjlement it. 
The ncn-halting of this program is easily detected by a system that 
knows that "recursive programs without stop rules don't stop". A more 
interesting situation is illustrated by the following sceraric. The 
program generator (human or mechar.ical) adds the line: 

5 IF :NUkBER=:0 S10P 
The program new counts down to 1 and halts. Ihe programmer then changes 
the program to make it countdown ly twos: 
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LO CCUKTLOWN :NUkEER-2, 

ard finds hiEfelf in a simple forr of a very typical tvg ritur tior- The 

detection of the r.on-halting rituatior is Ciore complex in this case, tut 

within the state c£ ajprojriate arts. 

The stejs in one approach are (1) recornize frca 
the structure of the prcprai: that it will always 
halt if and on3y if the ecuation K-2);=0 has a 
jositive integral solution fcf X for all positive 
integers K, and (2) show that K-2X=0 does not have 
a solution in integers when K is odd. 

Ke £ain confidence in the feasibility of the project by noting that 

these steps lie in areas (translation of fornialism and symbol 

E&nipulation) thai have shown firL progress. 

Fron a certain practical point of view this is 
enou£h. But our theoretical orientation leads to 
a dissatisfaction with an "unnatural", 

"Eatheffiatician's 11 formulation of the sitple fact 
that counting down ty twos car/t fet you frcm 
seven to zerc, 

A very different approach is illustrated ty 
elevating the concept of "conservation* 1 to the 
status of a fundamental primitive about which the 
systea cught to know. The bup under discussicn 
could then be recognized, as bv a person, through 
the qualitative knowledge tnat svbtractirg two 
conserves even-ness — rather than through the 
ranijulation of algelraic equations ! 

2.11 Goldstein's Prccrait-Understanding Irogran 

We have ^enticned one aspect of Ira Goldstein's thesis coal: to rake a 
program that will understand an object jrograc sufficiently to decide 
whether it will halt and if not, why not- Besides this generalized kind 
of understanding, Goldstein's program will understand the intentions of 
programs written lor a particular subject domsin cr arini-vcrld. This is 
in the sa^e spirit as Winograd'r program, which is able to understand 
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complex Erglif.h sentei.ces, provided that thete ere aboul a nini-world 
(the FLOCKS KORID) of which the program har thorough fcnovled/e. 
Gelds tern s mini-world consists of two-dimensional line-drawings, sich 
at' stick-men. his immediate jroblem is developing an spprcpriete 
description language lor this mini-world. This work draws on ideas from 
work en vision an<? is expected to contribute to it, mere rarticulrrlv as 
the robotics projects move towards more natural vision. 

Gcldstein's program-understanding-program will ceal with very simile 

Sphics programs (the "object prorram") written in LOGO, with the 
enticn of crawing a scene such as "men walking in file". The test of 
the program wall be its ability to diagnose semantic burs in the obiect 
program, ihe hind of knowledge required tc do this cculd be used in 

iLany application areas inducing: making a core intelligent teaching 
machine {to teach prof ramming) ; automatic debugging in Sussman's style;' 
automatic program writing. 
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Iart IIJ. CCKPLTER-COrr: 'BCUED VISICH AJ:D HAKIR'LATIOH 

■ 

Research en computer vision ard its aj-plicaticns js entering e nev* ph?se 
ir the Laboratory/. In the jast 9 the work centered t.rourd a variety of 
prototype protleEE and attempts tc develop sound theoretical models of 
those problems. We now feel that we have a very firm understanding of 
these problens at all levels and want to use this knowledge to move irto 
broader and mere tractical applications. 

In the past, vision research in I he Laboratory fecussed en the "BLOCKS 
WCRLD". This provided configurations of three-dimensional blocks* 

wedges, and other polyhedra simple enough tc prcvide access to basic 
probleEs yet complicated enough to ensure that the problem solutions 
will have general relevance. 

Even within the BLOCKS WORLD, it was necessary tc work out new fches.es 
of prcrrain organization, as well as new techniques fcr djfital picture- 
processing to be able to analyse scenes in which clearly delineated 
feometric objects occlude one another in three-dimensional space. The 
wcrld, however, is rot comjosed of uniformly illuminated geometric 
objects, and current research is pointed at the problercs of natural 
environments. 
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Cur applications research is being done in conjunction with 
work en the mini-robot cevelopraent project furded by a 
supplementary rropcsal* foork on ratural environment vision 
and manipulation is to include efforts on the followinc 
problems; all the resultr are to be available vjthir our 
modular "heterarchical vision system" described in pricr 
proposals and currently working at a demonstration level. 

3.1 Heterarchical Vicion Syslera and Project Direction 
p. Winston, B-K.P* Horn 

3.2 Generalization of Labelling Theories 
— — D. Waltz 

3.' Grouping, and Tactile Scone-Analysis 
T. Finin 

3.4 Analysis of Curved-Line Drawings 
__ _m; Adler 

3.5 Color Vision 

— — Km Lavin 

3*6 Touch and Tactile Programming 

: D. Silver 

3-7 "Low Level" Vision Programs 

J . Lercan 

3.8 Physical Knowledge and Stability 
— S. Fahloan 

3.9 liquids and Hand-Eye Tracking 

— __r. WoodhaE 

3.10 Electronic Absently; Another Robot "World" 
P. Winston and others 

3-11 Analysirg Complicated Objects 

J. Hollertach 

3.12 Groups, Descriptions, ard Conflicts 

— H« Dunleavy 
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T.I The Heterarchical Vision Syetei 

Work en problems of natural vision situations is rost clorely supervised 
by Prcf- P. Winston, whose personal concerns ere with the theory of 
scene-analyris in the EIACKS V.GRLE, automatic learning from experierce 
of meaningful three-di&ensional structures, the induction problem * or 
meaningful groups of objects, and the prcfcler.s cf cvercll system 
organization that arise because cf the variety of kinds of knowledge 
involved in "perception"* B*K.P. Horn is responsible fcr supervising 
applications research and development of image processing systems* 

3*2 Generalization cf Labelling Theory 

D* Waltz* recent thesis ties together much earlier vcrk done at cur 
laboratory and others on theory of geometric scene-analysis. The 
sequer.ee cf events was basically this: early work on the role of 
^T— joints" and other local features pointed the way to programs that 
cculd solve the "figure— ground" problem cf separating visual objects. 
This work reached a relatively successful but very complicated plateau 
with the vork of Guzman, who used a variety of formal heuristics for 
linking, grouping, and lirJc-inhibjting cues* 



The behavior of Guzman's program was better "explained*' with the 
introduction cf formal semantics for the linking heuristics ty Fufftan 
and Clowes- In that model, it was xecogrizec explicitly that different 
kinds of edges arise from diifexent classes of three-dimensional 
situations. Ai plying these ideas, and using a new rnalysir* of the 

situations in which Guzman's program encountered difficulties, Eattrer 
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ard ethers developed procedures that were simpler, more leaninpfYl, end 

n.cre effective than the older, more "ryntrctic" versions. In 

particular, the elaboration cf complicated "linJ-inp" heuristic* were 

replaced by s much simpler system of alternate linking rnd "splittiir" 
of objects f.part. 

In his thesis, Waltz carried further the analysis of the semantics of 
eoges, ard showed corclusively tiat in the riOCKS WCRLD there are may 
surprisingly strong constraints or the line drsvinrs derived by 
projection from three dimensional scenes. Cnce there constraints are 
uraerstcoo, ore ci:n often get directly at the rearing of each line ir a 
drawing, and identify it as concave or convex edge, obscuring edre, 
crack, shadow, or other sort of line. The techniques one' ir plications 
of this appeal- to have significance also in other semantic areas such as 
natural -Language and automatic prcgrarming. 

Waltz is now working to 2dd features to enatle the program to use mere 
global facts than were used in the thesis work, and is developing a 
graphic display program to facilitate study of the labelling process. 
He is supervising the work of Adlcr (see below). He hopes tc fird cut 
hew far his new labelling techniques can be adapted to work on curved 
objects, and poorly illuminated natural scenes. In particular, he is 

interested in the semantics of features of civilized interior scenes 

office, home, hospital, school, etc., and relations between verbal end 
visual scene— descriptions. 



3-3 Grouping and Tactile Scene- Analysis 

T. Finin is completing work en recopni2irg objects in contexts that ere 
complicated ty such serious ccclusiors that rot ruch of f-ome object can 
be seen. The methods depend on assuiainf that regularities consistent 
with the portions of the scone in direct view pre continued irto the 
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urseer portions. Such regularities right be detected, fcr example, by 
the heuristics propo/ed an P. Vina ten's thesis. Finin hopes tc carry 
this further into the systei: for hf>nd-eye coordination, rc that the 
conjectures about the unseen portions can be confiroed or rejected by 
tactile explorations. Ir the simplest version of such ar application, 
the robot would be profran^ed to completely disassemble the scere, 
operating at all times en -objects indirect view. Olviously, this 
should be retarded as a last resort; and there are cases in which any 
disassembly constrained to so operate would pass thorough a stage of 
instability in which the structure would collapse, damaging delict te 
parts. If the program's hypothetical reconstructed image of the three- 
dimensional situation is exploited, ore should usually be able to settle 
ambiguities by a very few carefully chosen delicate tactile probes that 
need not disturb stability, if the latter is taken into account in the 
selection. 

T. Iczaro-Perez is directly concerned with extending such grouping 
heuristics. In the BLOCKS tvORLD, Winston proposed grouring heuristics 

based on chains cf similar relations, and groups connected by conron 
relations to another object. Lozsnc-Perez is studying criteria for 
grouping hypotheses ir more complicated specialized domains, including 
the influences cf sizes, eclorr, textures, proximities and apparent 
functions. 
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7.4 Analysis of Curved-Line Drcvinfs 

Although a great deal is row known about analysing straight—line 
drawings that represent three— dimensional scenes, tuch less is kncvn 
about drawings with curved lines. Some prcj-osals have been Bade by 
Guzman and Huffman in earlier work. Current work, at Stanford, on 
scene-analyris of curved three-dimensional objects is t&Fed en surface 
ard range-finding data. Mark Adler, in our project, is working on 
extensions of the Guzman and Huffman ideas, to analyse drawings of 
outlines of three dicensional otjects that occlude one another in the 
viewer's image. 

Several problems nust be confronted immediately. To recognize curved 
otjects we need adequate descriptive laethods that are not too sensitive 
to position changes. One might begin by adapting Winston's feature and 
relation network descriptions to images with curved boundaries. 
Ambiguities are resolved by attempts to Dap local configurations onto 
models (as in an earlier proposal by Guzman that was never implemented). 
To do this effectively, however, the effects cf occlusion of features by 
otjects must be handled realistically, and the proposal is to use a 
sequence of analyses, beginning with use of T-^oints to detect 
occlusions, followed by conservative assumptions about intact unoccluced 
regions to serve as slarting units for the description matchirgs. 
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The procedure fcr matching coc els will probably fce an advance in 
sophistication, in the sense that each model will contain kncwlecge 
embedded within itself shout the effects of occlurion discoveries on 
what rext to look for* Thus, if in searching fcr a natch to a model of 
a jtan one fines that the cleeve oi an arm is occluded, one would rpecjfy 
that finding a "hard" in the right place is acceptable. (In the 
previous, knowledge- free experiments cf this sort, one would have had to 
find a reasonably clore fit to a continuation of the cleeve on the other 
side of the occlusion.) Eventually, such predictions could be deduced 
from three-dimensional ncdels, and we hepe to combine this work with 
that at Stanford en constructing such models* Adler plans to develop 
his jrocedures ueing the world cf cartoon drawing conventions*, before 
facing the more complicated situations that arise in pictures of real 
otjects. 



3*5 Color Vision 

K. lavin, ore of our new students, flans to investigate eoler theories 
such as those of Land, Lettvin, et al. in the context of the robot 
environment* This dimension cf visual experience has received 

relatively li ttle attention in rcboti cs* Implement ing a color 

specialist prcgrafr may also provide powerful tools for investigating the 
current theories of human color virion which, at the present ti^e. 



IAGE 37 



repair surpriringly ccntrcverr iel. For the Rcfcot itself, we lope to fet 
8 tetter and core useful uncerstanding cf hcu region chape, bcundtry 
sharpness, lighting chanfes, and ether perturbations influence programs 
embodying such theories. And finally, we would like to see vhetler 
there is any practical or ccmputaticnal advantage in using o 
"physiological theory" instead of a naive spectruE-characterizaticn. If 
the proximity of colored objects has substantial effects en nearby ones, 
which is quite likely the case, it cay well turn out that one of these 
theories, in which color-dercrif tiers tend to be relative to what 
happens en ir.ter-region louncaries rather than deperdinf completely on 
spectral content, might actually le of advantace to a robot. 

3*6 Touch and Tactile Protracting 

D- Silver has recently applied our six-axis force sensing wrist to turn 
cianks and screws on nuts, and plans to continue to work en applications 
that are difficult or impossible to do using vision alone. Such work 
has long been deferred in our Laboratory, even though it figured in the 
earliest proposal, because the problems of obtaining usable vision in a 
natural environment were so difficult. The time is now ripe for this, 
ar.d the results will be an important part of the facilities to be 
supplied to the mini-robot project's system. 
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3-7 "Low-Levt.l Vision" Programs 

Years of work have shown convincingly thst no pass-oriented single-ices 
line cr edge-finc'er can work in realistic environments. High-level 
driving programs must have available e variety cf specialists ior 
identifying sharp background liner, die Irtericr lines, shadows, cradle, 
reflections, highlights, etc. Detecting unexpected lines requires 
procedures that search cut from known vertices. Fringing together a 
wide variety of previous work, J. Lennan has concentrated on organizing 
a supply of primitives fcr such activities into a system that Is bettor 
integrated than ary of its predecessors. It will bring the L'altz ed^e- 
senantic labelling ideas into the very first steps of vision processing, 
by the way of constraints on partially labelled, partially analyzed 
fragments of scenes. These constraints serve to restrict places and 
types of lines to be looked for next. lermen hopes to take his system 
able to avoid areas congested with many short segments end close- 
tcgether potentially ambiguous vertices; such areas which shculd be 
left until the system has had a chance to apply mere global analysis of 
the three-dimensional situation to the long, clear lines ard isolated 
sharp vertices that are mere easily understood. 
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;.8 Physical Krowledge and Stability 

S. Fahlman is completing a ccnstruction-ilanr.ing procedure that couples 
a sophisticated understanding cf gravity and support principles with 
heuristics for coping with complicated construction tasks. Depending on 
the situation, his system may elect to build a structure one piece at a 
time, by prefabricating substructures, or by Indirection — using 
scaffolds or counterweights to provide temporary support. L'ia thesis 
will describe a "specialist* that contains advanced knowledge about such 
natters and the ability to propose plans for using such krowledge. 

3-9 Liquids 

R. Woocham is investigating li ttle-explcred prcblens in visual 

tracking, and is building up a rew experimental "world" that may turn 

out to be ideally suited for cur new steps toward practical 

applications. In his system the robot will pick up a container and 

pour a liquid (ceffee) into a cup. A liquid-pouring "specialist" will 

track the dark ecpe of the liquid as it moves in space and rises 

relative to the supposed edge of the cup. The following is a brief 

excerpt from his proposal, descriline the coffee-table environment: 

A Problem Domain For Studying Hand-Fye Coordination 

On a table, there are several coffee cups, a coffee pot, a 
bowl containing sugar cubes, a small pitcher of cream and e sroon 
or other object suitable for stirring. There is no oarticular 
arrangement to the objects on the table. They are randomly placed 
witnin the field oi view of a vidirsector eve and within the reach 
of a mechanical an—hand. 
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A humar engager in a rhort dialogue xequeFtirg a cud of 
coffee in any one of its standard configurrtiors (ie. black, 
creeiii, cream & sugar, rugar on3y, double cream, etc.). Tie arm- 
hard proceeds to c elect a Cup, pour the coffee from the pot, add 
the recuired eiibellishrentr and stir the result. The huiwn picks 
up his cup of coffee arid soys, "Thank you!* 

Features Of Such A System 

1) We would le demonrtrating a feneralized flexibility. Since 
there would be no specified arrangement of objects or the table 
nor a fized recipe for coffee, the robot would h^ve to both 
visually locate the objects and construct a plan ar reauired. 
Further, ve would like the systen to be general enough to allow 
for the addition/deletion of cups while the operation is in 
prcgress. 

2) We would be exhibiting a true hand-eye system in an environment 
realistically approaching that of the real world. In particular, 
the operation of pouring must accomodate a real world that charges 
dynamically, not just in discrete steps. Visual feedback, with 
the arm-hand in the visual field, would be an ersential 
prerequisite to accomplish accurate pouring. 

3) We would be exhibiting a somewhat generalized canirulative 
capability through the use of simple tcols — a pot fcr pouring and 
a spoon for stirring* 

4) We would te facing the issue of quality control. Visual 
feedback must certainly be used to monitor pouring. In addition, 
feedback must te ured to protect against pcurirg into a cup that's 
fallen over or pouring into a cup that's already full. Similarly, 
feedback must also be used to keep from knocking over a cup when 
stirring its contents. 

Is This A Gcod "Toy" System? 

The idea of a robot ceffee maker probebly strikes one at 
first as being a gcod demonstration* It certainly would te that. 
However, in considering possible alterrative rroblem dcnalrs for a 
hard-eye system, I believe that the robot coffee maker is also the 
most appropriate. 

The coffee Baker environment is rich enough to support the 
thcrough investigation and development of the various kinds of 
feedtack tools ard capabilities that would be required in any 
harxi-eye system. The processes, involved ir caking a cup of coffee 
are quite characteristic of the kinds of processes reouired in a 
generalized hard-eye system. 

The primitives recuired to meniter the risirg level of 
coffee in a cup are seen as essentially equivalent to thore that 
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would be required to careful! v ali^n the edfes cf objects In a 
corplex assembly procedure, "he primitives required to stir the 
contents of a cup with a spoon ere essentially equivalent to those 
that vould be required to tighter a rut vith a i.rench or turn a 
screw vith a screwdriver. Similarly, the primitives required to 
locate a cup J "or rouring are essentially equivalert to these that 
would be required to locate a hole for inserting a bolt or screw. 

Of equal significance, however, is the fact that the 
coffee iraker ervirenment is alro simple encuph to support ruch an 
investigation with a Kini&um arcourt of tifce required to deal with 
outside issues. I believe the current vision system cm easily be 
modified to handle the specific objects required for the coffee 
maker. In any event, I can immediately berin developing 

techniques for visual feedlack by restricting myself, for the time 
being, to polyhedral cups and jets. 

The coffee nakixig system involves an environment that is 
sufficiently dynamic so as to require a decree cf interaction that 
would constitute a significant advarce ever previous work in 
machine hand-eye coordination- The primitives developed for the 
coifee maker would be applicable to a host of other hand-eye 
tasks. At the same time, the coffee caker represents a problem 
domain that is very accessible and manageable given the current 
Status of the KIT VISICM SYSTEM 



3*10 Electronic Assembly: Another Robot "World" 

In our proposal for extension of our mini-robot development project , we 
describe a possible application tc electronic assembly, maintenance end 
repair* The objects are electronic components* Here agsin, is an 
ervirenrcent in which the visual problems are core complex than those in 
previous toy problems, but are believed to be within reach. 
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3-11 Analysing Complicated Object? 

J- hollerbach it caking £ood prefers ir carrying ur out of the domain 
of very simple geometric objects. In that worJd, identification end 
Eaniptlation problems are often "too easy" because blccks and wedges are 
completely defined and located once a few location and dimension 
ptxaceters are known. Heller bach has propored to develop descriptive 
rcethocs fcr complicated pclyhedra, with indentations and rrotrusiens, in 
which the descriptions are structured in levels of 'detail, caking 
location and identification reasonably convenient. At the sa^e tine, he 
is studying "simple" objects like bottles and cups with a view toward 
relating their deecriptiors to those cf ccoplex apprcxicatine polyhedra. 
Different approximations that are very similar in quality of dimensional 
fit can be vastly different in detcriptioral complexity, and the coal is 
to find techniques fcr describing complex otjects in natural ways that 
cake it easy to relate them to such prominent features as their glolal 
outlines and their decompositions into meaningful jarts (such as "toj", 
"handle", "lees", etc.)* 
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3.12 Groups, Description!-, ar.d Conflicts 

■ 

K. Dunieavy has projosec a project concerned with developing 
hierarchical models of hew such conplex structures as veils, windows, 
dcors, and chimr.eys can be cabined tc form builcinfs. There ere 
probleLS here that do net iirrcediateay t.eet the eye; the concept of 
"trick wz,ll" must be defined in a somewhat indeterminate way as a 
repeated ("eroui-like") structure with, usually, some boundary- 
termination condition. The tox cr hcuse concept of four vails ceeting 
tc form a box-like enclosure is easy to describe at a high level, tut 
ore has to engage additional knowledge to construct ore — because cne has 
to make cany decisions about what harpens at the intersections, where 
the kir.ds of descriptions that seem reasonable for the separate 
components may cone irto conflict. One can build a right angle in a 
brick wall without cutting any bricks in two, and if the vail is two 
layers thick, one can even make v.indcws without cutting. How is such 
knowledge to be represented? Dunieavy *s plans include heuristics for 
laying out the global forms, figurine out where the conflicts occur, and 
then applying local methods tc attempt to remove the conflicts. 
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Pert IV. Develcj-flier.t 0/ Research Methods ard Tcols 
4*0 Programmirg Languages for A.I. 

We have long felt that a r-ajor impediment to the progress of Artificial 
Intelligence is ihe 3ack of an appropriate lanfuage in which programs 
can be expressed J or the computer. Although the situatior har improved, 
an inordinate effort is still necessary in order to translate even a 
veil developed theoretical idea into an experiment in computer 
performance. And even when the experiment is finally carried out, mcst 
of the programming effort is ultimately wasted, because cf the eronacus 
difficulty in taking modifications tc tept new ideas sufgested ty it« 

Cur goal in this respect, is to develop computer languages in which the 
ccncejts and tethcds of A.I* can be expressed easily and translated by 
the computer into flexible, intelligitle end modifiable programs. 

This goal cannot be separated as a "service* project with any degree of 
independence from the mainstream of Artificial Intelligence Research. 
On the contrary, constructing such languages could be redescribed as 
developing a formalism and fundamental set cf primitive concepts for 
A.I. Thus, it is very close to the theoretical core cf research and, in 
our laboratory, language development has traditionally been integral 
with the substantive projects aiEed at particular p-erforaance goals, 
nevertheless, frofr time to time, issues in language design become 
crystallized froii the general research end acquire enough clarity and 
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Loaentum of their own to lecore separate research topics. 

We are very much at such a moiient in lime, She rest few years hns s^en 
a dramatic change in clarity and intensity of discussion rbout largua^es 
fcr Artificial Intelligence, This wave, we believe, befan in cur 
laboratory with the publication of early versions of the languoge 
RANKER (and the simplified dialect HlCHO-PLAIiHER), and has spread 
throufh the A.I. community internationally. She effect of this w*?ve vas 
that, for the first time, the issues related to computer language for 
A.I. were formulated in ccicmor terms so that experiences in meny 
projects and many places could be comjared more directly, In the pa^t, 
eech research project tended to develop its own ©d hoc extensions to 
general purpose languages such as LISP. 

Ar other interaction — from a surprising quarter — is addirg to the 
intellectual ferment about such issues. This is the interaction between 
the language problems for "serious A,I. programs* 1 and the problems that 
arise from our attempts to develop computer languages for the use of 
children in cur elementary education project. Experience and new 
theoretical insights make it increasingly clear that the languages we 
use, and plan to use, for children (collectively known as LOGC) need to 
develop towards greater and clearer expressivity for talking about 
fundamental heuristic ideas. But this is exactly what is needed also 
fcr Lachine intelligence! In the pest the work with children vas loo 
tentative for such parallels to surface and give rise to actual 
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interaction* This has charged, and now lerds to specific rucgestitns 

for cocji.or effort. The "language for children 11 is beinr re-exacined in 
extensive di^cusrionr within the I#b end rore widely* During the 
current year we eypect them to five rise tc the cesipi rnd 
implementation of at least one new cenputer lr.nfuare vhich will try to 
be clear enough for a child tc use (cuite literally!) ard powerful 
enough for research in Artificial Intelligence, 



4*1 PLANNER Progress 
— C- Hewitt and others 

4*2 Parsing Strategy Research 
V. Pratt 

4.3 Detugging Mds 

S. Markov itz 

4.4 Keterarchy 
E. Preuder 

4-5 Automatic Theorem Proving 
—.—.....— —^^— — A. Brown, A. Kevins. J* Ceiser 

4*6 Relaticn tc our PrcgraiL of Research en Education and Cognitive 
Develop Eert 
S. Itepert and others 



<*1 PWNNER Progress 

■ 
* * * 

PLANNEI.-like languages have now become widely accepted as 
important tools for research in Artificial Intelligence. The 
following crouxs are currently using such a tool: 
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Stanford University A.I. Laboratory 

Stanford Research Inrtitute A.I. Croup 

Carnegie-Hellon University 

Edinburgh University 

K.I.T. Artificial Intelligence laboratory 

Project t;AC Automatic Programing Grcup 

Ve feel that we ere nov in a period of consolidation for these new 

higher level formalisms. A variety of implemertation methods and 

ideas have teen tried. Recently we have developed a modular 

activation formalism which unifies FLAHNER-Uke languages and 

points to how they can be efficiently implemented. We propose to 

do the following: 



To sjend the next year completing an implementation of 
PLANNER based on this formalism. 

To investigate the fcr&al properties of the codel using 
the techniques develcy^d by Hewitt and Paterson for 
comparing powers of different control structures. 

To do a feasibility study of constructing a processor 
based on our model. This computer might execute 
FLANI,ER-like fornalisms 4CV60 times faster than a PDF- 
10. 

Cur applications research is being done in conjunction with 
Project MAC under a contract for the development of systems that 
do interactive logistics planning- These systems should mcke it 
easy to add more useful knowledge end after-thoughts. P. Eishop 
(hardware implementation), R. Steiger (formal properties of the 
model), Gary Peskin (data base hash coding), Gcrden Eeredict 
(knowledge packaging) are working with Hewitt en these plans. 
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^.2 Parsing Strategy Research 



Prof. V. Pratt is working on a linguistics Crier, ted Language, IIJIGCL, 
started at Stanford in 197C, which has been used to write e M deep 
structure 11 analyser, an Englich-to-German translator pre to type, a 
cenprehension and question-answering program. LIHGC-L implements 1he 
bookkeeping details of a context-free parsing algorithm thet provides 
fcr the intervention of user-supplied cede at each application of a 
ccntext-free rule, allowing the urer to provide semantic details cutside 
the scope of the context-free syntax. The program needs improvement in 
the parsing strategy, which currently has to consider all assignments of 
surface structure befcre choosing one. This is cauticus tut costly; it 
requires reveral thousand words of storage and several secords of 
processor tine fcr a 15-word sentence, but breeks down altogether for 
3C-word sentences. Giver the grammars in use, this cakes it unusable 
fcr such applications as reading newspapers, journals, thi^ sentence, 
and others where it Eight otherwise be adequate, Pratt plans to test 
two or three strategies to circumvent this protolea without complicating 
the user's prcblens. Pratt plans a conversatioral program that will 
start out as a version of the existing question arswering system and 
grow as deficiencies in its English are repaired; it cculd prcve of 
value to people vho need, a {ood English f rent-end for their programs, 
such as the one supplied by Woods to KASA lunar geologists, or to 
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linguists for other purposes. 



4.3 Debusing Aids 

S. horkcwitz is studying debugginc aids for the new languages embedded 
in LISP as meta-ltngu^ges, ar.d preparing a set cf programs to malyse 
files and prepare reports about function crors- references end 
itemizations of function-calling patterns and variables. The use of 
heterarchical goal-directed program invocation through; ptittern-mctching 
requires a new debugging technology, and we expect to develop one in a 
period shorter than was usual in the post, when the importance of such 
matters was rarely understood at language-developtent time. 



4-4 Heterarchy 

-■■ ' • 

As the results of other work cor.e to fruition, the early shortage of 

methods that worked minimally well is starting to be replaced by a 

richness of viable alternatives; this is what makes it reasonable to 

begin to attack practical problems in natural environments. E, Ireucer 

is committed to taking a new lock at our vision system, to plan a rew 

version of such a system, polarized around the notion of heterarchy from 

the very start. Freuder imagines knowledge available in this system to 

interact at all levels with mechanisms available in the new Planning 
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languages. 



A*3 Automatic Theorem Proving 



Although there is a tradition amcng iLathematiciars and nhilcrophers of 

attempting tc model "reasoning" as a font cf "logical" procedure, we 
maintain that there is a deep problem in the traditional atter.pt to 
separate the kincs of knowledge in the "data base" frot. the kinds of 
knowledge used to make plausible inferences thereof. But in order to 
understand the situation, we feel that we have tc thoroughly understand 
the strengths and limitations of the logistic method* We have several 
snail investifaticns aimed at what we believe are important questions in 
this area. 

A. Irown is studying the problem of automatic oerivaticn of proofs of 
theorems in abstract group theory, in an attempt to combine modern 
heuristic techniques, classical predicate calculus and the rew problem 
solving languages* 

A- Hevins is continuing his work on the same subjects, with emphasis on 
Leaking the rystem iiore intelligent about case anaHyses, etc* 

J* Ceiser if studying a variety of theories for representing empirical 
knowledge within classical logics, and is exploring theories of 
inference schemes for logical data bases that are knovn to possibly 
contain inconsistencies 
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4-6 Research or Education rnd Cognitive Development 

It is relevant at thir point to cocmert or the relaticn between our wcrk 
or* Machine Intelligence and en natural Intcllif ence. (Thir wcrk is 
funded tt&inlf by the K£F. ) We have net taken "singulation of hur-an 
thought processes" as a gcal, or even as a guiding principle cf the wcrk 
on Machine Intelligence. Rather, we have adopted a ro-hclds- barred 
approach to achieving performance, irrespective of whether the 
mechanises look like what might be happening in the human brain. 
However, we lind increasingly that the concepts which our wcrk on 
Machines forces us to adopt are applicable to Human thinking. They have 
led us to account for known phenomena in child development and to 
discover new phenomena* Most important, our procedural approach to 
thinking about thinking is distinguished froc that of traditional 
psychology in the simple and immediate way in which it translates into 
suggestions for educational practice* For example, our discussion of 
hew to formulate powerful, qualitative principles of ccrcmor— sense 
reasoning (using primitives such as particular conservation end general 
program-structural ideas) coulc be paralleled by a very rimilar 
discussion on how to formulate arpects cf scientific thinking usually 
classified as "intuitive 1 * and so usually epposed to "formr.!".* Dcing so 
also E.akes them mere teachable for exactly the same reason ar it maizes 
thee Lore programtable. Thus our approach to education involves a much 
mere radical attempt than usual to reccncepturlize whole areas of 
knowledge. 
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A particular area of educationally directed wcrk that has beer receiving 
growing attertior in the laboratory coulf: fc*> called "cuall tat: ve 
intuitive physics 1 ' • It is very rossifcle th at during the next year this 
work vill become rufficiertly mature to branch into prorrrrmirc projects 
directed at fivir.p EEChires the ability to "think physically" in a very 
general sense. Ve expect this wcrk to have practical corsequences, for 
exacple, in achieving the "pecple-trackiiif ■ applications. 



